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In this paper we generalize the concept of random networks to describe networks with non trivial 
features by a statistical mechanics approach. This framework is able to describe ensembles of undi- 
rected, directed as well as weighted networks. These networks might have not trivial community 
structure or, in the case of networks embedded in a given space, non trivial distance dependence of 
the link probability. These ensembles are characterized by their entropy which evaluate the cardinal- 
ity of networks in the ensemble. The general framework we present in this paper is able to describe 
microcanonical ensemble of networks as well as canonical or hidden variables network ensemble with 
significant implication for the formulation of network constructing algorithms. Moreover in the paper 
we define and and characterize in particular the structural entropy, i.e. the entropy of the ensembles 
of undirected uncorrelated simple networks with given degree sequence. We discuss the apparent 
paradox that scale-free degree distribution are characterized by having small structural entropy but 
are so widely encountered in natural, social and technological complex systems. We give the proof 
that while scale-free networks ensembles have small structural entropy, they also correspond to the 
most likely degree distribution with the corresponding value of the structural entropy. 
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INTRODUCTION 

The quantitative measure of the order present in com- 
plex systems and the possibility to extract information 
from the complex of interactions in cellular, technologi- 
cal and social networks is a topic of key interest in mod- 
ern statistical mechanics. The field of complex networks 
[l|, 0] has having a rapid development and a large success 
in this respect due to the wide applicability of simple 
concepts coming from graph theory. The characteriza- 
tion of the structure of different networks has allowed 
the scientific community to compare systems of very dif- 
ferent nature. Different statistical mechanics tools have 
been devised to describe the different level of organiza- 
tion of real networks. A description of the structure of 
a complex network is presently performed by measuring 
different quantities as (i) the density of the links, (ii) the 
degree sequence [H, (iii) the degree-degree correlations 
UM, 0, (iv) the clustering coefficient 0,11, (v) the k- 
core structure 

BEE!, (vi) the community structure 
@> 12. [l3l H13 and finally the nature of the embedding 
space |l5l. Il6l. Tl\. Moreover, is the network is weighted, 
strength/degree correlations [18| and if the network is di- 
rected, in-degree/out-degree correlations [l9[ are signifi- 
cant characteristics of the network These phenomenolog- 
ical quantities describe the local or non-local topology of 
the network and do affect dynamical models defined of 
them Q. 

While many different statistical mechanics models have 

13 to de- 



tion has been addressed to the study of entropy mea- 

o approach this prob- 



been proposed |20|, |21|, |2J, |2£ \2£ [25|, [2 



scribe how the power-law degree distribution can arise 
in complex networks, little work has been done on the 
problem of measuring the level of organization and " or- 
der" in the frame of theoretical statistical mechanics. 
Only recently, in the field of complex networks atten- 



sures _ , , , 

lem. In 3l| the entropy of a given ensemble as the nor- 
malized logarithm of the number of networks in the en- 
semble has been introduced. This quantity can be used 
to asses the role that a given structural characteristics 
have in shaping the network. In fact, given a real net- 
work, a subsequent series of randomized networks ensem- 
bles can be build each subsequent ensemble sharing one 
additional structural characteristic with the given net- 
work. The entropy of these subsequent networks ensem- 
bles would decreases as we proceed adding constraints 
and the difference between the entropies in two subse- 
quent ensembles quantifies how restrictive is the intro- 
duced additional constraint. In the first part of this pa- 
per construct a general statistical mechanics framework 
for the construction of generalized random network en- 
sembles which satisfy given structural constraints. We 
call these ensembles " microcanonical" . We also describe 
how to construct "canonical" network ensembles or gen- 
eralized hidden variable [H, 0, 0, ^ models. 
Subsequently we make an account of most of the network 
ensembles that can be formulated: the ensemble of undi- 
rected networks with given number of links and nodes, 
the ensemble of undirected networks with given degree 
sequence, with given spatial embedding and community 
structure. Some of these network ensembles where al- 



ready presented in 3l| and we report their derivation 
here for completeness. This approach is further extended 
to weighted networks and directed networks. Finally we 
focus our attention on the structural entropy, i.e. the 
entropy of an ensemble of uncorrelated undirected sim- 
ple networks of given degree sequence. The structural 
entropy of a power-law network with constant average 
degree is monotonically decreasing as the power-law ex- 
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ponent 7^2. This result could appear in contradiction 
with the wide occurrence of power-law degree distribu- 
tion in complex networks. Here we show by a statistical 
mechanics model that scale-free degree distribution are 
the most likely degree distribution at given small value 
of structural entropy while Poisson degree distributions 
are the most likely degree distribution of networks with 
maximal structural entropy. 

This result indicates that the scale-free degree dis- 
tributions emerges naturally when considering networks 
ensembles with small structural entropy and therefore 
larger amount of order. 

The appearance of the power-law degree distribution 
reflects the tendency of social, technological and espe- 
cially biological networks toward "ordering" . This ten- 
dency is at work regardless of the mechanism which is 
driving their evolution that can be either a preferential 
attachment mechanism J3|, or a "hidden variables" mech- 
anism EHHiS' 

mechanics mechanism [2C 



E of the network ensemble is defined as 




or some other statistical 



STATISTICAL MECHANICS OF NETWORK 
ENSEMBLES 

A network of N labeled nodes i = 1,2,..., TV is 
uniquely defined by its adjacency matrix a of matrix el- 
ements > with > if and only if there is a 
link between node i and node j. Simple networks are 
networks without tadpoles or double links, i.e. an = 
and = 0,1. Weighted networks describe heteroge- 
neous interactions between the nodes and the matrix ele- 
ments aij can take different null or positive values, while 
directed networks are described by non-symmetric adja- 
cency matrices a ^ a T where we have indicated by a T 
the transpose of the matrix a. 

A structural constraint on a network can always be 
formulated as a constraint on the adjacency matrix of 
the graph, i.e. 



F(a) = C. 



(1) 



=0V(iJ) 



(3) 



The marginal probability for a certain value of the ele- 
ment a^ of the adjacency matrix is given by 



^{A) = y^Siaa - A)S (F(a)-C 



The probability of a link Py is given by 
dlnZ 



Pij 



dhi 



(4) 



(5) 



=0V(i,j 



In and ensemble of weighted network we can define also 
the average weight Wij of a link between node i and node 
j as equal to 



dlnZ 



(6) 



fc«=r«=0V(t,j) 



In a "microcanonical" network ensemble all the networks 
that satisfy a given structural constraint have equal prob- 
ability. Therefore the probability of a network G, de- 
scribed by the adjacency matrix a, is given in the "mi- 
crocanonical" ensemble by 



Pm (a) 



-JVS ! 



F(a) - C 



(7) 



If we allow for "soft" structural constraints in network 
ensemble we can describe "canonical" network ensemble. 
In a " canonical" network ensemble each network a has a 
different probability given by 



(8) 



expression that for ensemble of simple networks take the 
form 



(9) 



In order to describe "microcanonical" network ensem- 
bles with given structural constraints in [3l| and in the 
following we will use a statistical mechanics perspective. 
Therefore wc define a partition function Z of the ensem- 
ble in the following way 



Z = V S [#(a) - C 



(2) 



where, for simplifying the problem ,-F(a) and a^ take 
only integer values, and S[-] indicate the Kronecker delta 
and Q(x) = 1 if x > and Q(x) — if x — 0. Moreover, 
in (|79|) . the auxiliary fields hij have been introduced as 
in classical statistical mechanics. The entropy per node 



If the link probabilities 7Tjj (ay ) are chosen equal to (2]) 
and ([5]), then we have that the structural constraints 
-F(a) = C are satisfied in average, i.e. 



(F(a)) Pc(a) = C 



(10) 



where the average Op^a) indicates the average over the 
canonical ensembles ©. The statistical mechanics for- 
mulation of network ensemble is always well defined. For 
network structural constraints that do not correspond to 
feasible networks [34[ the entropy of the network ensem- 
ble is nevertheless E = — 00. Although the definition of 
the statistical mechanics problem is always well defined, 
the calculation of the partition function by saddle point 
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approximation can only be performed if the number of 
constraints F a with a = 1, . . . , M is at most extensive, 
i.e M = 0(N). In addition to that, in the paper we 
are going to consider only linear constraints on the ad- 
jacency matrix. Further developments on this statistical 
mechanics framework will involve pertubative approach 
to solve non linear structural constraints. 

UNDIRECTED SIMPLE NETWORKS 

In an undirected simple network the adjacency matrix 
elements are zero/ one (a^ = 0, 1) and the tadpoles are 
forbidden ( an = Vz). We can consider for these net- 
works different types of structural constraints. In the 
following we list few of them of particular interest. 

• i) The ensemble G(N, L) of random networks with 
given number of nodes N and links L = J2i<j a ij 
(providing in this way a statistical mechanics for- 
mulation of the G(N,L) random ensemble). In this 
case we have the structural constraint 

F(a) - C = ^ «y - L = C 11 ) 

• ii) The configuration model, i.e. the ensemble of 
networks with given degree sequence {k\, . . . , kn} 
with ki = ■ . In this case the structural con- 
straints are given by 

F Q (a) - C a = a ^ ~ k » = ( 12 ) 

3 

for a= 1, ...,N. 

• Hi) The network with given degree sequence 
{fci, . . . , fcjv} and given average nearest neighbor 
connectivity k nn (k) = [J2ij^( k i ~ k)a i:j kj]/(kN k ) 
of nodes of degree k (with N k indicating the num- 
ber of nodes of degree k in the network). In this 
case the structural constraints are given by 

F a {a) -C a = Y, a<*j ~k a = (13) 

3 

for a — 1 , . . . , N and 
F a (a) - C a =J25(h - fydijkj - kN k k nn (k) (14) 

for a = N + 1, . . . N + K. with K indicating the 
maximal connectivity of the network. 

• iv) The network ensemble with given degree se- 
quence and given community structure. For these 
network we assume that each node is assigned 
a feature {qi, . . . qn} and we fix the number of 
links between nodes of different features A(q, q') = 



Ei<j ~ _ <l') a v with <fe = minfe, qj) 

and ~q~ij = max ((^, q 3 ). In this case the structural 
constraints are given by 

F a (a) -C a = Y, a aj -k a =0 (15) 

3 

for a = 1 , . . . , N and 

F a (a)-C a = Sjqij - q)$(qij ~ q')aij 

i<j 

-A(q,q') (16) 

for a = N + 1, . . . N+ Q(Q + l)/2 with Q equal to 
the number of different features of the nodes. Here 
an in the following in order to have an extensive 
number of constraints we assume Q = 0(N 1 / 2 ). 

• v) The ensemble of networks with given degree se- 
quence and dependence of the link probability on 
the distance of the nodes in an embedding geomet- 
rical space. In this ensemble we consider fixed spa- 
tial distribution of nodes in space {r±, . . . , f^} and 
we consider all the networks compatible with the 
given degree sequence and the number of links link- 
ing nodes in a given distance interval. Therefore 
we take A distance intervals Ii = [de,di + (Ad)?] 
with £ — 1, . . . , A, and we fix the number of links 
linking nodes in a given distance interval. The 
structural constraint involved therefore the vector 
B{di) = J2i <3 Xe(di,j)a,ij where dij = d(f f i,fj) is 
the distance between node i and j in the embedding 
space and the characteristic function xi( x ) = 1 if 
x G [df, di+(Ad)i] and Xd(%) — otherwise. In this 
case the structural constraints can be expressed as 

F a (a) -C a = Y, a aj -k a =Q (17) 

3 

for a = 1 , . . . , N and 

F a (a) ~C a = J2 Xt{di,j)a i:j - B{d t ) (18) 

i<j 

for a = JV+1,...JV + A. 

The G(N,L) and the G(N,p) ensembles 

The networks in the G(N, L) ensemble have given num- 
ber of nodes N and links L. The entropy of this ensemble 
is given by the logarithm of the binomial 

/ N(N-l) \ 

iVS = f 2 j (19) 

(we always assume distinguishable nodes in the networks 
[29(). The probability pij of a given link is given 

by = L/(N(N — l)/2) for every couple of nodes i, j. 
The ensemble G(N,p) is the "canonical" ensemble corre- 
sponding to the " microcanonical" G(N,L) ensemble. 
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The configuration ensemble 

In the configuration ensemble we consider all the net- 
works with given degree sequence. Using ([2]) (fT2|) the 
partition function of the ensemble can be explicitly writ- 
ten as 

Zi = e n - e a «) eEi< ' hijaij (2°) 

{ay} » j 



in the "canonical" model the connectivity of each node 
is distributed according to a Poisson distribution with 
average J2j Pij- 

The form of the probability pij is such that when infer- 
ring the values of the " hidden variables" u>i for a " canon- 
ical" network in this ensemble by maximum likelihood 
methods, we obtain the lu[ = u>i in the large network 
limit El- 



Expressing the delta's in the integral form with La- 
grangian multipliers u>i for every i = 1 , . . . N we get 

Z x = f Vuj e-^' UJ ' k *Y[(l + e UJ ' + ^ +h ^) (21) 
■> i<j 

where Vuj — ]X du>i/(2ir). We solve this integral by 
saddle point equations accounting also for second order 
terms of the expansion. The entropy of this ensemble of 
networks can be approximated in the large network limit 
N > 1 with 



NY.'' 



and 



+]Tln(l + e^J) 



(22) 



with the Lagrangian multipliers u>i satisfying the saddle 
point equations 



_! 1 

and the coefficients q; defined as 



E 



(23) 



(24) 



The probability of a link i, j in this ensemble is given by 



Pg 3 



UJ: +U1 ■ 



1 



(25) 



In particular in this ensemble ftj ^ f{u>i)f{uj), con- 
sequently rte model retains some "natural" correlations 
[27| given by the degree sequence and the constraint that 
we consider only simple networks. In fact these are noth- 
ing^ else than the correlations of the configuration model 

The " canonical model" corresponding to the configura- 
tion model is then a "hidden variable" models where each 
node i is assigned a "hidden variable" u>i and the proba- 
bility for each link follow (|25|) . Similar expressions where 
already derived in different papers [13, 3, 27| but with 
a different interpretation. Here the "hidden variables" 
u>i are simply fixing the average degrees of each node. 
We note here that the derivation of [26[ guarantees that 



Uncorrelated networks 



The case in which there is a structural cutoff in the net- 
work fcj < yj (k)N is of particular interest. In this case we 
can approximate Eq. (|23|) by e u>i ~ fci/y (k)N, a>i ~ fcj. 
In this limit the network is uncorrelated the probabilities 
of a link are given by p^' uncorr = kikj/((k)N), since 
the w* < 0. We call the entropy of these uncorrelated 
ensembles the structural entropy Eg and we can evaluate 
it providing the explicit expression 

NY S ~ -^ln[fc. i / v /(fc)]v]fc,-i^ln(27rfc 4 ) 

i i 

1 ^ — ^ k^kj ^ — ^ 1 k^ kj 



(fc)7V ^4((fc)7V)2 
^(lnfc 4 -l)fc 2 -i^ln(2 7 rfc l ) 



i{k)NMW)-i]-\(yQ 



.(26) 



Expression (|26[) gives for the number of networks in 
the ensemble 



((k)N)\ 



exp 



(k) 



■OQnN) 



(27) 

From combinatorial arguments we can derive an ex- 
pression j\f^ ncorr for the number of uncorrelated net- 
works with a given degree sequence which agrees with 
the above estimate (|27|) in the limit N ^> 1, i.e. 



OQnN). 



(28) 



In fact by combinatorial arguments we can show that the 
number of networks with given degree sequence is given 
by the following expression in the large N limit, i.e. 



'uncorr 



(2L - 1) 



(29) 



The factor (2L — 1)!! accounts for the total number of 
wiring's of the links. In fact if we want to construct a net- 
work, given a certain distribution of half-edges through 
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the N nodes of the network, as a first step we take a half- 
edge and we match it with one of the 2L — 1 other half- 
edge of the network. Secondly we match a new half-edge 
with one of the 2L — 3 remaining half-edges. Repeating 
this procedure we get one out of (2L — 1)!! possible wiring 
of the links. This number includes also the wiring of the 
links which gives rise to networks with double links. To 
estimate the number of such undesired wiring we assume 
that the network is random, i.e. that the probability that 
a node with fej half-edges connects to a node with kj half- 
edges is a Poisson variable with average hkj/ ({k)N). In 
this hypothesis the probability II that the network does 
not contain double links is equal to [36] 



n 



n 

i<3 



(k)N 



<fc> 



(30) 



Finally in the expression (|29[) for Af c there is an additional 
term which takes into account the number of wiring of 
the links giving rise to equivalent networks without dou- 
ble links. This term is given by the number of possible 
permutation of the half-edges at each node, i.e. fli^" 
We note here that a similar result was derived by mathe- 
maticians for the case in which the maximal connectivity 
K < N 1 / 3 13711 and an inequality was proved for the case 
K > TV 1 / 3 [38|] . Now we extend these results by statis- 
tical mechanics methods to uncorrelated networks with 
maximal connectivity K < -J (k)N. 



The entropy of a network ensemble with fixed 
degree correlations 



We consider now network ensembles with given de- 
gree correlations and given average degree of neighboring 
nodes, satisfy the constraints defined in Eqs. (flU]) and 
(H3J). We can proceed to the evaluation of the probabil- 

(2) 

ity of a link p^' and the calculation of the entropy of 
the ensemble as in the configuration model. In this case 
we have to introduce the Lagrangian multipliers fixing 
the degree of node i and the Lagrangian multipliers Af. 
fixing the average degree of nodes of degree fc. 

The partition function of this ensemble can be eval- 
uated at the saddle point giving for the entropy of the 
ensemble, in the thermodynamic limit value 



und 



i k 

Vhi(l + e u "+ u i +kiA ^+^ A ^i) 

i<j 

I^ln(2 TOl )" i$>(2™ fe ) (31) 



where ui* and A s k tar satisfy the saddle point equations 



^u}*+uj*+kjA* k .+kiA k . 



k = V 

1 + e uj i +u 'i +k ^ A k +k ' A "^ 



(32) 



k nn (k) — S(ki k) kj 

UVfc - j¥ 4 1 + e 



u* +kj A^. +k i A k '^ 



u?+uij+k : jA k '_+kiAl _ 



and where with cti,ak are approximately equal to the 
following expressions 



on ~ 



i 1 + e 



U 'i -^fe. . 



(33) 



o t»)J +kj A% . +ki A% . 



L — ' J /, u'+u'+kjAt +kiAZ V 



( 2) 

The probability p- of the link in this ensemble is 

ui*+u*.+kjAl+kiA kj 

A 2 ) _ _~ 



given by 



1 + e 



(34) 



This formula generalize the " hidden variable" formula of 
the configuration model to networks with strong degree- 
degree correlations. In particular in order to build a 
"canonical" network with strong degree degree correla- 
tion we can consider nodes with "hidden variables" 9i 
and Gg and a probability pij to have a link between a 
node i and a node j given by 



Pij 



0i0j(G Bi ) e >{G 9i ) Bi 
l + e^iGe^iGg^ 



(35) 



The entropy of network ensemble with given degree 
sequence and given community structure 

The partition function of network ensembles with given 
degree sequence (TT5|) and given community structure (TIB)) 
can be evaluated by saddle point approximation in the 
large network limit as long as Q — 0(N 1 ^ 2 ). 

Following the same steps as in the previous case we 
find that the entropy for such an ensemble is given by 

i q<q' 

+ ^m(l + e^ + ^ + ™* ( V*« ) " 

i<j 

-i^ln^)-^ 1 ^™^) (36) 

i q<q' 

with the Lagrangian multipliers {ojf ar }, {w^T} satisfy- 
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ing the saddle point equations 



E 



A(q,q>) = 



(37) 



q ) x 



i<j 



e u*+ujj+w* (q,q') 

V 

^ i gW?+oJ+tu*(9,?') ' 

and with a^, a g „/ that can be approximated by 



E 



o ^*+^*+iu*(9. 3 , 9ij ) 



(38) 



*<3 



u]*+u>*+w* {q,q) 



In this ensemble the probability for a link p\- between a 
node i and a node j is equal to 

u;*+w*+w*(o ,(?;-) 



P 



(c) _ 



1 + e' 



(39) 



Assigning each node a "hidden variable" #i and to each 
pair of communities the symmetric matrix V(q, q') we can 
construct the "hidden variable" or "canonical" ensemble 
by extracting each link with probability 



= 0,0 X [<,,.<,,) 

Pv l + e^Viq^qj) 



(40) 



The entropy of a network ensemble with given 
distance between the nodes 



Finally we consider the ensemble of undirected net- 
works living in a generic embedding space and with struc- 
tural constraints described by (fT7| and [)18p. Following 
the same steps as in the previous cases we find that the 
entropy for such an ensemble in the large network limit 
is given by 



52kiu>t -Y,B(dt)g(d t )* 

i (=1 
^ m _|_ e "i+"j+£( Xi( d a)g* (di)^ 
i<j 

1 1 A 

-^ln(2 TOl )--^ln(2vra £ ) (41) 



with the Lagrangian multipliers {uJi}, {g<i} satisfying the 
saddle point equations 

ki = E 7 7 e <+^+E f xK^)9*(^) ^ 42 ^ 

S(d/) = E»(^)7— ^ 
»<i J- + e * 



<+S*(<W) ' 



and the variables ai,a q , q > approximated by the expres- 
sions 



E 



p ut+ u j+T,t Xe(dij)g* (dt) 



Of 



^Xd{d %] )- 



p ui*+uj*+g* (d t ) 



(43) 



to 



^1 + e w * +"i+f*( df )^ 
The probability for a link between node i and j is equal 



P 



Id) _ 



e <4+w|+s*(d*) 



(44) 



Therefore the " hidden variable" model associated to this 
ensemble correspond to a model where we fix the " hidden 
variables" 0i and W{dg) and we draw a link between node 
i and node j according to 



(45) 



WEIGHTED NETWORKS 

Many networks not only have a non trivial topological 
structure but are also characterized by weighted links. 
We will assume in this paper that the weight of a link 
can assume only integer values and consequently a link 
between a node i and node j is characterized by an integer 
number > 1, this is not a very stringent constraints 
since we can assume to have always finite networks (stud- 
ied in the thermodynamic limit). In a weighted network 
the degree and the strength Sj of the node i are defined 
as 



(46) 



£=1 



where 0(x) = if x = and Q(x) = 1 is x > 0. It is pos- 
sible to define series of weighted networks by considering 
networks with fixed total strength, with given strength 
sequence, with given strength and degree sequence and 
proceeding by adding additional features as in the un- 
weighted case. Here an in the following we study the 
most relevant cases: 
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• We first consider the network ensemble with given 
total strength S. The structural constraint in this 
case is equal to 



F(a) - C = ^2 Oij ~ s = 0. 



(47) 



l<3 



• ii) We consider the network with given strength 
sequence si, . . . , sjv- The structural constraints are 
for this ensemble given by 



F(a) a - C a = a aj - s a = 0. 



(48) 



for a = 1,...,N. 



• iii) Finally we consider the network ensemble with 
given degree sequence {fci, . . . , k n } and strength se- 
quence {si, . . . , sn}. For this ensemble the struc- 
tural constraints are given by 

F(a) a - C a = 6(a ai ) - k a = 0. (49) 



for a = 1, . . . , iV and 

F(a) a - C a = ^ a aj - s a = 0, 

j 

for a = 1,.. .,2N. 



(50) 



The entropy of weighted network ensembles with 
given total strength S 

The entropy of this ensemble is given by 



NEY = In 



m^ 11 + s 

N(N-l) 



The average value of a the weight of the link from i to j 
is given by 



W; 



j..\W 



N(N-l) 
2 



(51) 



and the probability of a link between node i and j is 
equal to 



w,i 
Pa = 



S 



S- 



N(N-l) ' 



(52) 



Therefore the simple networks with adjacency matrix 
((Aij)) that can be constructed from the weighed net- 
works with adjacency matrix ((aij)) by putting Aij — 
Q(dij) is uncorrelated. The canonical ensemble is 
given by Eq. ([8|) with 



TTij (a%j ) 



1 - e u 

andw = - ln[l + N(N - 1)/(2S)]. 



(53) 



The entropy of weighted network ensembles with 
given strength sequence 

To calculate the entropy of undirected networks with 
a given strength sequence of degrees {si} we proceed by 
the saddle point approximation as in previous cases We 
find that the entropy of this ensemble of networks is given 
by 



(54) 

with the Lagrangian multipliers io\ satisfying the saddle 
point equations 

E U>* +UJ* 
r^r. (55) 



and with Ai being the eigenvectors of the Jacobian of the 
function 



^=5>[i- 



— UJi —UJ-i 



(56) 



i<j 



The average value of a the weight of the link from i to j 
is given by 



w 



1 - e 



(57) 



and the probability of a link between node i and j is 
equal to 



W,l u>*+u* 



(58) 



Therefore as it has been observed in [? ] only by rewiring 
the links of a network allowing for multilinks we get a 
network structure which is uncorrelated. 

The canonical ensemble © in this case can be con- 
structed by assigning to every possible link (i,j) the 
weight with the probability 

Kij (aij ) = ; z^TL^ ■ ( 59 ) 



1 -e 



The entropy of weighted network ensembles with 
given strength /degree sequence 

The entropy of weighted networks with a given 
strength and degree sequence {sj,fcj} in the large size 
network limit is given by 



TVS 



w 



1 + e^+^i 



4E 27V E ln (2^A,) 



(60) 
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with the Lagrangian multipliers satisfying the saddle 
point equations 



V 

j£ (e^+^l + e -K+<4) - l)(e^^* - 1) 



161) 



(62) 

and with being the eigenvectors of the Jacobian of the 
function 



1 + e^ 1 " 1 "'^ ■ 



1 



(63) 



calculated at the values {oj*, ^>*}. The average weight of 
the link (r/) is given by 



-K+w* )+o*+</>*) 



(e 



l)(e" 



-1) 



(64) 



and the probability of a link between node i and j is 
equal to 



W.2 



Pi 



(65) 



The canonical ensemble (j8]) in this case con be con- 
structed by assigning to every possible link the 
weight ay with the probability 

^ ) = ; , i • ( 66 ) 



1 + e 



A _ Jr 



for a = 1 , . . . , N and 

F(a) a - C a = aja ~ C = 0. (69) 

3 

for a = N+ I,..., 2N. 



The entropy of directed network ensembles with 
fixed number of directed links 



If we consider the number of directed networks A/g 
with given number of nodes and of directed links we find 

<*=( JV( £,7 1) ). (T») 

In this case the probability of a directed link is given by 

L 



Pij 



N(N - 1)' 



(71) 



The entropy of directed network ensembles with 
given degree sequence 

To calculate the entropy of directed networks with a 
given degree sequence of in/out degrees {fc°"*,fc™} we 
just have to impose the constraints on the incoming and 
outgoing connectivity, 

{a,ij} i j i j 

expfy^ hjjajj] (72) 



DIRECTED NETWORKS 

An undirected network is determined by a symmetric 
adjacency matrix, while the matrix of a directed network 
is in general non-symmetric. Consequently the degrees of 
freedom of a directed network are more than the degrees 
of freedom of an undirected network. In the following we 
only consider the network ensemble with 

• i) Total number of directed links The structural 
constraint in this case is equal to 



F(a)-<7 = 5^oy - 5 = 0. 



(67) 



• ii) Given directed degree sequence 
{kf ,kl kp\k^ ut) }. The structural 

constraints in this case are 



F(a) Q - C Q 



V n ■ - k° 



0. 



(68) 



Following the same approach as for the undirected case, 
we find that the entropy of this ensemble of networks is 
given by 



+ ^ln(l + e w *- 
-Wm^T^^) (73) 



with the Lagrangian multipliers satisfying the saddle 
point equations 



S out ) 



.(»«) _ 



2^ j e u,*+u,* 



3^1 



+ e 3 % 



(74) 
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with 



a 



(out) 



S in ) 



U^ 1 

r 

3¥ 



e a 1 



^ (1 + e w J +fi> «*) a 



(75) 



The probability for a directed link from i to j is given 



by 



P 



l,dir) 



l + e 



(76) 

N the directed 

l,(rfir) 



If the + u)j < (M,j = 1, 
network becomes uncorrelated and we have p, 

k\° wt 'kj I^J (kin)N. Given this solution the condition 
for having uncorrelated directed networks is that the 
maximal in-degree K^ m ' and the maximal out-degree 
K {out) ^ould satisfy K^K^ / y/(k in )N < 1. The 
entropy of the directed uncorrelated network is then 
given by 



ivZj l,d<r 



2 (ki n ) \kout) 



(77) 



which has a clear combinatorial interpretation as it hap- 
pens also for the undirected case. 



NATURAL DEGREE DISTRIBUTION 
CORRESPONDING TO A GIVEN STRUCTURAL 
ENTROPY 

For power-law networks with power-law exponent 7 £ 
(2,3) the entropy of the networks with fixed degree se- 
quence Ei given by Eq. (j2"2")l decreases with the value 
of the power-law exponent 7 when we compare network 
ensemble with the same average degree [3l| . Therefore 
scale-free networks have much smaller entropy than ho- 
mogeneous networks. This fact seems to be in contrast 
with the fact that scale-free networks are the underlying 
structure of a large class of complex systems. The ap- 
parent paradox can be easily be resolved if we consider 
that many networks are the result of a non-equilibrium 



dynamics. Therefore they do not have to satisfy the max- 
imum entropy principle. Nevertheless, in order to give 
more insight and comment on the universal occurrence 
of power-law networks in this section we derive the most 
likely degree distribution of given structural entropy when 
the total number of nodes and links are kept fixed. By 
structural entropy we define the entropy Ss (|26p °f un ~ 
correlated networks with fixed degree distribution. In 
order to do that we construct a statistical model very 
closely related to the urn or "ball in the box" models 
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We consider degree distributions {N k } = 5(k — hi) 
which arise from the random distribution of the 2L half- 
edges through the N nodes of the network. The number 
of ways jV{jv fc } m which we can distribute the (2L) half- 
edges in order to have a {N k } degree distribution are 



{N k } 



(2L)\ 



U k (kN k )\ 



(78) 



We want to find the most likely degree distribution that 
corresponds to a given value of the structural entropy. 

Proceeding as in standard statistical mechanics, we de- 
fine a normalized partition function Z as 



2 = ^T,^i^ mSsiiNk}) - 



(79) 



{N k } 



with C = (2L)!exp[/3(2L)!!]. The role of the parameter 
(3 in Eq. ([79]) is to fix the average value of the structural 
entropy S5. When (3 — > 00 the structural entropy S5 is 
maximized when (3 — > /3 m i n the structural entropy Eg is 
minimized. 

In equation (|79| the sum over the {N k } distribu- 
tions is extended only to {N k } for which the total number 
of nodes N and the total number of links L in the network 
is fixed, 



i.e. 



k 

kN k = 2L. 



(80) 



To enforce these conditions we introduce in (|79p the delta 
functions in the integral form providing the expression 



1 f dX f f da f dv \ 



-p^N k lnk\- 



P ( s 



4 \(k) 



Y,H(kN k y. 



i\(2L - Y N k k ) ~ wi N -J2 N k)~ iv(NS - ^ k 2 N k ) 



(81) 
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z = 



. dX f dfi f dv 
dS I to to /2^ eXP 



i\2L - ifiN - ivNS - j (jr?) + ^lnG fc (A, /jt, v) 



4 \{k) 



(82) 



where 



G k (\,n,v) 



kN k 



iX + i^- + ii/fc — — Inffc!) 
fc fc 



(83) 



Assuming that the sum over all N k can be approximated 
by the sum over all L k = kN k = l,2,...oo we get 



In Gfe(A, /Li, i/) = exp iX + i^i/fc — t ln(fe!) + ivk 

f3 ( S 



f(X,n,y, S) 



and 
4 \{k)) 

k— ^ ln(Jfe!)+'M / Jfe (84) 



i{k)X — ifi — ivS 

-Y, 



where < fc >= 2L/N indicates the average degree of the 
network. By evaluating (|82p at the saddle point, deriving 
the argument of the exponential respect to A and v, we 
obtain 



1 v- 1 i 
N^k 

k 



X+iju/fe-f ln(fc!))+ii/fc 



,iA+i/i/fe-f ln(fc!))+ii/fc 



— N fc 2 e ^+Wfc-f ln(fc!)+ii/fe 



AT 



ivN = -f] 



S 



2{kf 



.(85) 



These equations always have a solution for sparse net- 
works with L = O(JV) provided that > 1 and (fc) > 1. 
The marginal probability that = kNk is given by 



P(L k = kN k ) = 



1 



-/3N k (ln(k\)+i\k+in+ivk 2 ) 



(kN k y. 

Z k {L,kN k ,N) 

x W) • 



(86) 



with 

Z k (L,£,N) -- 
and 



If we develop (fB"6"|) for I <C L and we use the Stirling 
approximation for factorials, we get that each variable 
L k is a Poisson variable with mean (L k ) satisfying 



(L k ) 



= {N k ) — k-f 3 - 1 e iX +f 3 + i L L / k + iuk 



(89) 



where we assume that the minimal connectivity of the 
network is fc > 0. The average {Nk) is a power- law dis- 
tribution with a lower and upper effective cutoffs — ifi 
and l/(w) fixing the average degree (fc), with the La- 
grangian parameter A fixing the normalization constant 
and finally fixing the structural entropy. The distribu- 
tion of P{N k ) is finally 



P(N k ) 



(kN k y. 



-PN k \n{k\)+i\kN k +i)iN k +iii(k)' 2 



(90) 



In the limit (3 — > oo ([9H]) is extremely peaked around 
the average degree fc ~ fc* = 0({k)) of the network and 
the degree distribution Nk decays at large value of Nk as 
a Poisson distribution, i.e. 



P(N k ) 



1 



(kN k )\ 



kN k [-(3 ln(fc* !)/fe* +iA/c* + i\i /fc* +ii/fc*] 



(87) 

- ^/JV)A - ifx(l - i/(kN)) + 

2 x 2 



In 



(91) 

. Therefore for /3 — > oo the network is Poisson like. In the 
opposite limit of small structural entropy and (3 small the 
P(Nk) distribution (|90p develops a fat tail decaying like a 
power-law (|89l) with an exponent 7 = {3+1. Therefore the 
natural distribution with a small value of the structural 
entropies are decaying as a power law and and smaller 
values of the power-law exponent correspond to a smaller 
value of the structural entropy. When the value of the 
entropy is minimal, (3 — > 1 the degree distribution (pJTj) 
has a large tail with an exponent 7^2. 



CONCLUSIONS 

In conclusion we have shown that there is a wide set 
of network ensembles that can be naturally described by 
statistical mechanics methods. The statistical mechan- 
ics method provides the theoretical estimation of the en- 
tropy of these ensembles that quantify the cardinality 
of the network ensembles. We believe that the entropy 
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of randomized ensembles constructed from a given real 
networks will be of great applicability for inference prob- 
lems defined on technological social and biological net- 
works. In this paper we have focused on some theoreti- 
cal problems that can be approached with the use of this 
quantity. First we have formulated a series of "canoni- 
cal" or "hidden variables" models that can be used for 
generating networks with community structure and spa- 
tial embedding. Secondly we have focused on the de- 
gree distribution of network. The degree distributions 
are not all equivalent. In fact the associated structural 
entropy depends strongly on the distribution. In partic- 
ular the power-law degree distribution with exponent 7 
and fixed average degree are associated to a structural en- 
tropy that decreases with 7. Nevertheless we have shown 
that power-law degree distributions are the more likely 
distributions associated to small structural entropy. This 
shed light on the evidence that power-law networks con- 
stitute a large universality class in complex networks with 
a non trivial level of organization. 

This work was supported by 1ST STREP GENNETEC 
contract No. 034952. 
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